feat(audit): default-on integration drift detection#1137
feat(audit): default-on integration drift detection#1137danielmeppiel merged 10 commits intomainfrom
Conversation
- Add _ReadOnlyProjectGuard context manager (utils/guards.py): snapshots stat of protected paths, raises ProtectedPathMutationError on any mutation. Defense-in-depth above the scratch-root remap. - Add CATEGORY_DRIFT + drift() recording method to DiagnosticCollector. - Add drift_count property and _render_drift_group renderer that groups by kind (modified/unintegrated/orphaned) with stable section header for machine consumers. - Tests: 7 unit tests covering happy path, mutation, creation, deletion, missing-tolerated, exception-not-masked, single-file protected path. Refs #1071. Phase A of WIP/drift/06-final-plan.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the drift detection feature per WIP/drift/06-final-plan.md (closes #1071 scope alignment with #898). Engine (Phase B): - src/apm_cli/install/drift.py: ReplayConfig, DriftFinding, CheckLogger, CacheMissError, normalization helpers (build-id strip, line endings, BOM), run_replay() (cache-only), diff_scratch_against_project(), text/json/sarif renderers, atexit scratch cleanup. - src/apm_cli/install/services.py: scratch_root kwarg with ensure_path_within defense-in-depth guard for replay isolation. - src/apm_cli/policy/ci_checks.py: _check_drift() wrapper returning (CheckResult, list[DriftFinding]); graceful CacheMissError handling. CLI surface (Phase C): - src/apm_cli/commands/audit.py: --no-drift opt-out flag with mutex against --strip/--file via UsageError. Drift wired into both _audit_ci_gate (--ci) and _audit_content_scan (bare project audit) paths, default-on per ADR-02. JSON/SARIF/text renderers integrated; --no-drift warning gated to text mode (stdout cleanliness). Tests: - tests/unit/install/test_drift.py: 13 unit tests (normalization, diff cases, renderers). - Legacy --ci tests opt out of drift via batch --no-drift injection (fixture parity, not a behavior change). 7597 unit tests pass; lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the locked test matrix for issue #1071 drift detection. Floor of 43 tests across three new files closes the 'ULTRA HARDENING OF HELL' coverage requirement. New files: - tests/integration/test_drift_check.py (32 tests): * Section A: 9 drift cases (modified/unintegrated/orphaned + CRLF/ BOM/Build-ID false-positive guards) * Section B: 4 past-PR regressions (#1067, #882, #889, source-deleted) * Section C: 7 edges (no/corrupt lockfile, untracked governed, no-write contract, idempotency) * Section D: 3 multi-target (copilot/claude/cursor) * Section E: 9 default-on / --no-drift opt-out (mutex, stderr routing, JSON suppression) - tests/integration/test_drift_check_e2e.py (10 tests): full install->mutate->audit loop with mix_stderr=False, air-gap proof, JSON/SARIF stability, 30s smoke - tests/unit/install/test_drift_perf.py (1 test): 100 primitives replay+diff under 5s Engine fix surfaced by tests: - src/apm_cli/install/drift.py: run_replay now reads apm.yml's target field via parse_target_field and passes it to resolve_targets. Without this, multi-target projects (copilot+claude+cursor) replayed only the auto-detected primary target, falsely reporting secondary target deployments as orphaned. Helper _read_apm_yml_target() added. CI wiring: - scripts/test-integration.sh: two new blocks in run_e2e_tests() invoking the integration + e2e suites before the final success log. Both safe to run without GITHUB_APM_PAT (cache-only, mocked network). Verification: 56 drift-domain tests pass; full repo lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- CHANGELOG.md: Added [Unreleased] entry under Added describing the default-on drift detection in apm audit, the three failure modes it catches, false-positive guards, --no-drift opt-out + mutex semantics, and the JSON/SARIF integration shape. Closes #1071, supersedes #898. - docs/src/content/docs/guides/drift-detection.md (NEW, sidebar order 7): Full user-facing guide -- what drift means, how the cache-only replay works (with mermaid diagram), exit-code matrix, when to use --no-drift, output formats, and the CI single-line gate that replaces the legacy git status --porcelain script. - packages/apm-guide/.apm/skills/apm-usage/commands.md: Extended the audit row with --no-drift flag and added a paragraph documenting the drift-by-default behavior, three failure modes, false-positive normalization, and JSON/SARIF integration. Aligns the skill that ships in apm-guide with the new CLI surface (per apm-keep-docs-up-to-date.instructions.md rule 4). - .github/workflows/ci.yml: Annotated Gate B (legacy bash drift check) with a comment marking it redundant once apm-action ships a CLI with default-on drift detection (this PR's release). Kept as defense-in-depth fallback until then. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 3 | 2 | Solid replay-engine design with frozen dataclasses and defense-in-depth guards; scratch_root kwarg is a pragmatic seam but warrants a follow-up to formalize as a deploy-target factory. |
| CLI Logging Expert | 0 | 1 | 3 | Drift output follows ASCII/stderr conventions well; missing actionable fix hint violates 'Include the fix' rule; verbose scratch-path disclosure absent. |
| DevX UX Expert | 0 | 2 | 2 | Missing recovery hint in drift output and missing cli-commands.md update; otherwise the surface design is sound. |
| Supply Chain Security Expert | 0 | 3 | 2 | Cache-integrity verification absent from replay path; scratch containment is sound; air-gap test blocks subprocesses but not socket-level egress; atexit cleanup is SIGTERM-fragile on shared CI runners. |
| OSS Growth Hacker | 0 | 3 | 2 | Strong launch-beat story; needs cross-links and a 60-second demo path to avoid the new guide being an island. |
| Doc Writer | 0 | 5 | 2 | Drift guide is well-written; ship after fixing two stale companion docs and one self-contradictory section. CHANGELOG bullet violates Keep-a-Changelog one-line rule. |
| Test Coverage Expert | 0 | 2 | 0 | 42 integration+e2e tests cover all three drift kinds, no-write contract, air-gap, SARIF/JSON stability, and --no-drift mutex. Two gaps: no test that normalization guards do NOT mask real drift, and CacheMissError path untested at unit tier. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 5 follow-ups
- [DevX UX Expert] Add recovery hint to drift output: 'Run: apm install' after each drift group -- Two-persona convergence (cli-logging + devx-ux). Violates 'Include the fix' rule. One-line fix that converts a diagnostic into a product moment. Should land in-PR.
- [Doc Writer] Update cli-commands.md (--no-drift flag), ci-cd.md (remove obsolete bash workaround), and quick-start.md (stale cross-ref) -- Three stale doc surfaces contradict the new behavior. Shipping without sync means day-one user confusion. Cheap fixes, same PR.
- [Supply Chain Security Expert] Track follow-up issue: verify cache content matches lockfile resolved_commit before replay trusts it -- Shared CI caches and mounted volumes are a real poisoning vector. Current defense (prior-install trust) is pragmatic for v1 but the integrity primitive (commit-SHA pinning) is bypassed at replay time.
- [Test Coverage Expert] Add inverse-normalization unit test asserting real content drift is NOT suppressed by BOM/CRLF/Build-ID guards -- Evidence outcome=missing on a secure-by-default safety invariant. A normalization bug could silently mask real drift with no automated guard catching the regression.
- [OSS Growth Hacker] Add inbound cross-links from ci-policy-setup, governance-guide, and making-the-case to the new drift-detection guide -- Guide currently has zero internal-link equity -- it's discoverable only via sidebar. Cross-links from high-traffic pages drive adoption of the new capability.
Architecture
classDiagram
direction LR
class ReplayConfig {
<<ValueObject>>
+project_root Path
+lockfile_path Path
+targets frozenset~str~ | None
+cache_only bool
}
class DriftFinding {
<<ValueObject>>
+path str
+kind str
+package str
+inline_diff str
}
class CacheMissError {
<<Exception>>
}
class CheckLogger {
<<Base+Subclass>>
+replay_start()
+diff_start()
+replay_complete(n)
+clean()
+findings(n)
}
class CommandLogger {
<<Base>>
+verbose bool
+error(msg)
+warning(msg)
+success(msg)
}
class _ReadOnlyProjectGuard {
<<ContextManager>>
+project_root Path
+protected_roots list~Path~
+__enter__()
+__exit__()
}
class ProtectedPathMutationError {
<<Exception>>
}
class DiagnosticCollector {
<<Collect-then-render>>
+drift(msg, path)
+drift_count int
}
class integrate_package_primitives {
<<IOBoundary>>
+scratch_root Path | None
}
CheckLogger --|> CommandLogger : extends
_ReadOnlyProjectGuard ..> ProtectedPathMutationError : raises
class ReplayConfig:::touched
class DriftFinding:::touched
class CacheMissError:::touched
class CheckLogger:::touched
class _ReadOnlyProjectGuard:::touched
class ProtectedPathMutationError:::touched
class integrate_package_primitives:::touched
classDef touched fill:#fff3b0,stroke:#d47600
flowchart TD
A["apm audit (commands/audit.py)"] --> B{--ci flag?}
B -- yes --> C["_audit_ci_gate()"]
B -- no --> D["_audit_content_scan()"]
C --> E{"--no-drift?"}
D --> F{"--no-drift AND no --file/--strip/--package?"}
E -- no --> G["[NET] _check_drift(project_root, lockfile)"]
F -- no --> G
E -- yes --> H["skip drift, warn on stderr"]
F -- yes --> H
G --> I["run_replay(ReplayConfig, CheckLogger)"]
I --> J["[FS] _make_scratch_root() tempfile.mkdtemp + atexit"]
J --> K["_assert_scratch_bound(project, scratch)"]
K --> L{"for lock_dep in lockfile"}
L --> M["[FS] _materialize_install_path() resolve cache path"]
M --> N["_build_package_info() loads apm.yml if present"]
N --> O["[FS] integrate_package_primitives(scratch_root=scratch_root)"]
O --> P["[FS] ensure_path_within(project_root, scratch_root)"]
P --> L
L -- done --> Q["diff_scratch_against_project()"]
Q --> R["[FS] _walk_managed(scratch, governed_roots)"]
Q --> S["[FS] _walk_managed(project, governed_roots)"]
R --> T["_normalize() on each file pair (BOM + CRLF + Build-ID strip)"]
S --> T
T --> U["emit list~DriftFinding~"]
U --> V{"format?"}
V -- text --> W["render_drift_text()"]
V -- json --> X["render_drift_json()"]
V -- sarif --> Y["render_drift_sarif()"]
sequenceDiagram
participant User
participant AuditCmd as commands/audit.py
participant CIChecks as policy/ci_checks.py
participant Drift as install/drift.py
participant Services as install/services.py
participant FS as Filesystem
User->>AuditCmd: apm audit [--ci]
AuditCmd->>CIChecks: _check_drift(project_root, lockfile)
CIChecks->>Drift: run_replay(ReplayConfig, CheckLogger)
Drift->>FS: mkdtemp (scratch dir)
Drift->>Drift: _assert_scratch_bound(project, scratch)
loop each locked dependency
Drift->>FS: _materialize_install_path (cache lookup)
Drift->>Services: integrate_package_primitives(scratch_root=scratch)
Services->>Services: ensure_path_within(project_root, scratch_root)
Services->>FS: write primitives to scratch
end
CIChecks->>Drift: diff_scratch_against_project(scratch, project, lockfile)
Drift->>FS: _walk_managed(scratch) + _walk_managed(project)
Drift->>Drift: _normalize() each pair
Drift-->>CIChecks: list[DriftFinding]
CIChecks-->>AuditCmd: (CheckResult, findings)
AuditCmd->>User: render_drift(findings, format)
Recommendation
Land the recovery hint and doc-sync fixes in this PR (both are trivial, same-day work), then merge. Track cache-integrity verification and inverse-normalization test as follow-up issues. The feature is architecturally sound, the test suite is strong (43 new tests), and the positioning value is high. No reason to delay beyond the in-PR polish pass.
Full per-persona findings
Python Architect
- [recommended] scratch_root kwarg on integrate_package_primitives leaks test-double / audit concerns into the production API surface at
src/apm_cli/install/services.py:96
Adding scratch_root to the production install function's signature couples a drift-audit implementation detail into the core pipeline. Cleaner seam: DeployTarget protocol injected by callers. - [recommended] Normalization helpers are drift.py-private but will be needed by compile, install verify, and future audit modes at
src/apm_cli/install/drift.py:96
_strip_build_id, _normalize_line_endings, _strip_bom belong in utils/normalization.py with a public API. - [recommended] _read_apm_yml_target couples drift module to manifest format at
src/apm_cli/install/drift.py:334
Tight coupling to apm.yml schema; pragmatic today, but a 'load project config' helper is the right seam. - [nit] render_drift_json return type annotation could be more precise
TypedDict or dict[str, list[dict[str, str]]] gives downstream callers type-safe access. - [nit] atexit.register may leave temp dirs on SIGKILL
Use tempfile.TemporaryDirectory context manager pattern for both atexit AND explicit cleanup.
CLI Logging Expert
- [recommended] render_drift_text and _render_drift_group omit an actionable remediation hint telling the user how to fix detected drift at
src/apm_cli/install/drift.py:640
Message Writing Rule Add ARM64 Linux support to CI/CD pipeline #4 'Include the fix'. Drift findings show WHAT but never HOW. Append 'Run apm install to re-sync deployed files with the lockfile.' - [nit] CheckLogger emits stderr progress unconditionally -- no quiet-mode or non-TTY suppression path
- [nit] _inline_diff_for never produces an actual diff for sub-cap files; docstring is misleading
- [nit] Verbose mode discloses tracemalloc peak but not scratch tmpdir path -- agent debugging is harder
DevX UX Expert
- [recommended] render_drift_text never tells the user HOW to fix drift -- the single remediation command is missing at
src/apm_cli/install/drift.py
Failure mode is the product. npm audit prints 'run npm audit fix'; APM should print 'Run: apm install'. - [recommended] docs/src/content/docs/reference/cli-commands.md not updated -- PR is incomplete per doc-sync Rule 4 at
docs/src/content/docs/reference/cli-commands.md
Persona scope: 'If a CLI change is not reflected in cli-commands.md in the same PR, that change is incomplete by definition.' --no-drift not in flags table. - [nit] --no-drift mutex error could name the reason more concretely
- [nit] CHANGELOG entry is a single monolithic paragraph -- harder to scan than bullet points
Supply Chain Security Expert
- [recommended] Drift replay trusts apm_modules cache without verifying content matches lockfile resolved_commit at
src/apm_cli/install/drift.py:231
_materialize_install_path checks only EXISTS, never that content/git HEAD matches lockfile.resolved_commit. A poisoned cache (CI runners with shared caches, mounted volumes) would produce false-clean. Security model says commit-SHA pinning is the integrity primitive; replay bypasses it. - [recommended] Air-gap test blocks subprocess binaries (gh/curl/wget) but not socket-level Python network calls at
tests/integration/test_drift_check_e2e.py:84
Patches subprocess.run/Popen but not socket.socket. Future requests/urllib egress would not be caught. - [recommended] atexit-based scratch cleanup is not SIGTERM/SIGKILL safe; sensitive deployed content may persist on shared CI runners at
src/apm_cli/install/drift.py:142
Python atexit doesn't fire on SIGKILL; CI cancellation = SIGTERM+SIGKILL. Scratch /tmp persists with replayed primitives. - [nit] inline_diff field could leak secrets if expanded -- add SECURITY comment
- [nit] Positive finding: drift replay correctly avoids credential paths (zero token references; NotImplementedError on cache_only=False)
OSS Growth Hacker
- [recommended] Drift-detection guide is an island -- no inbound links from ci-policy-setup, governance-guide, or making-the-case
Guide links OUT but nothing links IN. Zero internal-link equity. - [recommended] CHANGELOG entry buries the hook in implementation detail -- rewrite lead sentence as the one-liner users can repost
- [recommended] No 60-second demo path for adopters -- the guide explains but never invites the reader to try
- [nit] README 'Content security' bullet mentions audit but not drift -- missed hook
- [nit] Guide does not name the comparable ecosystem tools (npm audit, terraform plan -detailed-exitcode)
Auth Expert -- inactive
PR does not touch any auth-relevant file; drift replay is cache-only with no downloader/clone/token surface -- verified via grep + drift.py read. Network-enabled replay is explicitly gated by NotImplementedError.
Doc Writer
- [recommended] integrations/ci-cd.md still documents the bash git-status workaround that this PR explicitly supersedes at
docs/src/content/docs/integrations/ci-cd.md:61
Lines 61-79 still recommend the legacy pattern under 'Verify Deployed Primitives'; 'We dogfood this' callout still cites the old ci.yml. Most important doc-sync gap on the PR. - [recommended] reference/cli-commands.md does not document --no-drift and still describes --ci as 7-check-only at
docs/src/content/docs/reference/cli-commands.md:462 - [recommended] quick-start.md sends new users to the now-obsolete drift section in ci-cd.md at
docs/src/content/docs/getting-started/quick-start.md:159 - [recommended] drift-detection.md case Integrate copilot runtime #2 in 'When to use --no-drift' contradicts the CLI behavior described two paragraphs later at
docs/src/content/docs/guides/drift-detection.md:67
Lines 67-69 list 'Strip-mode invocations' as a reason to use --no-drift, but audit.py auto-skips drift in strip/file mode AND raises UsageError if combined. Self-contradiction. - [recommended] CHANGELOG entry violates Keep-a-Changelog 'one line per PR' rule
- [nit] New page is not back-linked from ci-policy-setup.md or governance-guide.md
- [nit] sidebar order: 7 collides with dev-only-primitives.md and org-packages.md
Test Coverage Expert
- [recommended] No test asserts normalization guards (BOM/CRLF/Build-ID) do NOT mask real content drift at
tests/unit/install/test_drift.py
Tests prove cosmetic-only changes are suppressed. None asserts the inverse -- the safety invariant. A future normalization bug could silently mask real drift.
Proof (missing at):tests/unit/install/test_drift.py-- proves: Normalization guards suppress only cosmetic changes, never real drift [secure-by-default,governed-by-policy] - [recommended] CacheMissError raised by run_replay has no direct unit test exercising the raise paths at
tests/unit/install/test_drift.py
Defined and raised in 5 locations; only command-layer swallowing is tested. No direct pytest.raises assertion.
Proof (missing at):tests/unit/install/test_drift.py-- proves: Cache-miss paths in drift replay raise CacheMissError so the audit command can catch and handle gracefully [devx,governed-by-policy]
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
CEO panel recommended landing two in-PR follow-ups before merge: 1. Recovery hint in drift output (cli-logging + devx-ux convergence): render_drift_text now appends '[i] Run apm install to re-sync deployed files with the lockfile.' so users see WHAT and HOW in one message. Honors Message Writing Rule #4 'Include the fix'. 2. Doc-sync (doc-writer + devx-ux convergence): - reference/cli-commands.md: add --no-drift to audit options table; amend --ci description to mention drift contribution. - integrations/ci-cd.md: replace bash 'git status --porcelain' workaround under 'Verify Deployed Primitives' with 'apm audit --ci' one-liner; update 'We dogfood this' callout text. - getting-started/quick-start.md: retarget stale cross-ref from the now-superseded ci-cd anchor to the new drift-detection guide. - guides/drift-detection.md: drop the self-contradictory case #2 in 'When to use --no-drift' (strip-mode is auto-skipped, not opt-out). - CHANGELOG.md: compress verbose entry to one Keep-a-Changelog line pointing readers to the guide for detail. Tracked as follow-up issues (CEO call): - supply-chain: verify cache content matches lockfile resolved_commit before drift replay trusts it (commit-SHA pinning bypass on shared CI caches). - test-coverage: inverse-normalization unit test asserting BOM/CRLF/ Build-ID guards do NOT mask real content drift (safety invariant). Lint clean. 45 drift tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addressed in-PR (commit 7907865)Per the CEO panel recommendation, landed the two convergent in-PR follow-ups: 1. Recovery hint (cli-logging + devx-ux): 2. Doc-sync (doc-writer + devx-ux):
All 45 drift tests pass; lint clean. Tracked as follow-up issues (CEO call)
StatusPR is ready for review. CEO stance: |
There was a problem hiding this comment.
Pull request overview
This PR extends apm audit to perform integration drift detection by default by replaying the install/integration pipeline into a scratch directory (cache-only) and diffing the result against the working tree. It adds a new drift engine, wires drift into text/JSON/SARIF outputs, updates CI/docs, and introduces extensive unit/integration/e2e coverage.
Changes:
- Add a cache-only “install replay + diff” drift engine with renderers (text/JSON/SARIF) and normalization guards (Build ID / CRLF / BOM).
- Wire drift detection into
apm audit(default-on) with--no-driftopt-out, and propagate findings into CI JSON + SARIF payloads. - Add read-only defensive guard + substantial test coverage; update CI scripts and docs to reflect the new drift surface.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Bumps apm-cli version to 0.12.1. |
| src/apm_cli/install/drift.py | New drift replay + diff engine with normalization + renderers. |
| src/apm_cli/install/services.py | Adds scratch_root kwarg + safety guard for replayed integrations. |
| src/apm_cli/utils/guards.py | New read-only project-tree guard used as defense-in-depth for drift. |
| src/apm_cli/utils/diagnostics.py | Adds drift diagnostic category + rendering. |
| src/apm_cli/policy/ci_checks.py | Adds _check_drift() to run replay+diff and return findings. |
| src/apm_cli/commands/audit.py | Adds --no-drift, runs drift by default, integrates drift into JSON/SARIF/text output paths. |
| tests/unit/utils/test_guards.py | Unit coverage for the read-only guard behavior (create/modify/delete). |
| tests/unit/utils/init.py | Ensures tests.unit.utils is a package for imports/discovery. |
| tests/unit/test_audit_policy_command.py | Updates audit policy unit tests to disable drift (stabilize expectations). |
| tests/unit/test_audit_ci_command.py | Updates CI-mode audit unit tests to disable drift (stabilize expectations). |
| tests/unit/test_audit_ci_auto_discovery.py | Updates auto-discovery unit tests to disable drift. |
| tests/unit/install/test_drift.py | Unit tests for drift normalization, diff kinds, SARIF rule IDs, and stderr-only phase logging. |
| tests/unit/install/test_drift_perf.py | Performance smoke test for replay+diff under a 5s budget for 100 primitives. |
| tests/integration/test_drift_check.py | Integration tests covering drift kinds, edges, multi-target behavior, and --no-drift mutex. |
| tests/integration/test_drift_check_e2e.py | E2E tests for no-write contract, air-gap behavior, performance smoke, and JSON/SARIF shape stability. |
| scripts/test-integration.sh | Ensures new drift integration/e2e suites run in CI integration job. |
| packages/apm-guide/.apm/skills/apm-usage/commands.md | Updates apm-guide command reference for drift default + --no-drift. |
| docs/src/content/docs/reference/cli-commands.md | Documents --no-drift and drift behavior in apm audit --ci. |
| docs/src/content/docs/integrations/ci-cd.md | Replaces legacy git status drift gate with apm audit --ci. |
| docs/src/content/docs/guides/drift-detection.md | New/updated guide describing drift behavior, formats, and CI usage. |
| docs/src/content/docs/getting-started/quick-start.md | Updates link/reference to drift checking guidance. |
| CHANGELOG.md | Adds Unreleased entry for default-on drift detection. |
| .github/workflows/ci.yml | Notes legacy bash drift step as redundant once apm-action picks up this CLI version. |
Copilot's findings
Comments suppressed due to low confidence (2)
src/apm_cli/policy/ci_checks.py:470
_check_drift()resolves targets viaresolve_targets(project_root)(auto-detect). If a project declarestarget:inapm.ymlbut the corresponding deployment dirs don't exist yet, auto-detect can omit those targets and the diff will miss drift (e.g., unintegrated outputs under.claude/,.cursor/, etc.). Consider resolving targets with the same explicit target selection used byrun_replay()(read fromapm.yml) so governed roots are complete even when dirs are absent.
logger.diff_start()
resolved_targets = resolve_targets(project_root)
if targets:
resolved_targets = [t for t in resolved_targets if t.name in set(targets)]
findings = diff_scratch_against_project(scratch, project_root, lockfile, resolved_targets)
docs/src/content/docs/guides/drift-detection.md:94
- The JSON example shows
driftas a list, but the implementation/tests treat the top-leveldriftkey as an object containing adriftlist (i.e.,payload['drift']['drift']). Please update this example (or adjust the implementation) so the documented schema matches the actual output shape.
**JSON** -- the audit report gains a top-level `drift` key:
```json
{
"report_format_version": "1.0",
"checks": [...],
"drift": [
{
"path": ".github/instructions/foo.md",
"kind": "modified",
"package": "<local>",
"inline_diff": "..."
}
]
}
</details>
- **Files reviewed:** 22/24 changed files
- **Comments generated:** 8
| def _snapshot(paths: list[Path]) -> dict[Path, tuple[float, int] | None]: | ||
| """Capture (mtime_ns, size) for each path, or ``None`` if missing. | ||
|
|
||
| Symlinks are followed; missing paths record ``None`` so they may | ||
| legitimately remain absent without triggering the guard. | ||
| """ | ||
| snap: dict[Path, tuple[float, int] | None] = {} | ||
| for p in paths: | ||
| try: | ||
| st = p.stat() | ||
| snap[p] = (st.st_mtime_ns, st.st_size) | ||
| except FileNotFoundError: |
| passed=False, | ||
| message=( | ||
| f"drift replay aborted: {exc} -- " | ||
| "run 'apm install' first or use --no-cache (not yet supported)" |
| def _check_drift( | ||
| project_root: Path, | ||
| lockfile, | ||
| targets=None, | ||
| cache_only: bool = True, | ||
| verbose: bool = False, | ||
| ) -> tuple[CheckResult, list]: | ||
| """Replay the install in a scratch dir and diff against the project. |
| for stream_name, stream in (("stdout", result.stdout), ("stderr", result.stderr)): | ||
| text = stream or "" | ||
| for ch in text: | ||
| assert ord(ch) < 128 or ch in {"\u2500", "\u2501"} or ord(ch) > 0xFFFF or True, ( |
| # Drift severities: kinds of divergence from the lockfile-defined state. | ||
| DRIFT_MODIFIED = "modified" # tracked file content changed | ||
| DRIFT_UNINTEGRATED = "unintegrated" # tracked file missing from project | ||
| DRIFT_ORPHANED = "orphaned" # untracked file present in managed dir |
|
|
||
| ### Added | ||
|
|
||
| - **`apm audit` now detects integration drift by default.** Read-only cache-only install replay catches missed `apm install` runs, hand-edited deployed files, and orphaned files. Findings exposed in JSON (`drift` key) and SARIF (`apm/drift/<kind>` rules); in `--ci` mode they contribute to the exit code. Opt out with `--no-drift` (mutually exclusive with `--strip`/`--file`). See the [Drift Detection guide](docs/src/content/docs/guides/drift-detection.md) for details. (#1071, supersedes scope of #898) |
| | `apm audit` | Reported in stdout | 0 (advisory only) | | ||
| | `apm audit --ci` | Reported and counted as failure | 1 | | ||
| | `apm audit --no-drift` | Skipped entirely | governed only by other checks | | ||
|
|
||
| In `--ci` mode drift findings are pooled with the seven baseline lockfile | ||
| checks (`lockfile-exists`, `ref-consistency`, etc.) -- a single | ||
| non-zero exit covers all of them. |
| @@ -593,6 +670,10 @@ def _audit_content_scan( | |||
| all_findings = [f for ff in findings_by_file.values() for f in ff] | |||
| exit_code = 1 if ContentScanner.has_critical(all_findings) else 2 | |||
|
|
|||
| # Drift findings escalate exit code to 1 (critical). | |||
| if drift_failed and exit_code == 0: | |||
| exit_code = 1 | |||
|
|
|||
…gnostics Bare 'apm audit' is advisory (exit 0 on drift); 'apm audit --ci' is the gate (exit 1). Closes the regression introduced when content-scan escalation accidentally also escalated drift findings. Also addresses inline review: - A2: vacuous ASCII-encoding assertion now scopes per-line - A4: tuple[float, int] -> tuple[int, int] in guards.py - A5: type-annotated _check_drift signature - A6: clarified DRIFT_ORPHANED comment - A7: CHANGELOG references PR + closes - A3: CacheMiss message now drift-specific (no --no-cache confusion) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per oss-growth: surfaces drift detection alongside content security and lockfile integrity in the conversion-critical Production-grade section, so a reader scanning for 'why APM' sees the supply-chain story end-to-end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
apm install drops a .apm-pin JSON marker into each cached package
root recording the resolved_commit; apm audit verifies it before
running drift replay. Catches the 'teammate bumped lockfile, did
not reinstall' + 'shared CI runner reused stale apm_modules'
scenarios that would otherwise silently produce misleading drift
output.
LockfileBuilder syncs markers UNCONDITIONALLY (even when the
lockfile YAML is unchanged and even when no install happens), so
existing users self-heal on their next 'apm install'.
This is stale-cache detection, NOT cryptographic integrity --
defending against active cache tampering requires content-addressed
hashes, which is deferred.
Schema (v1): {schema_version: 1, resolved_commit: <sha>}
Marker file: <install_path>/.apm-pin
Coverage:
- 14 unit tests in test_cache_pin.py (positive + every error path
+ skip rules + idempotent re-run + self-heal regression)
- 1 integration test in test_drift_check_e2e.py exercising the
full install -> mark -> verify flow against a synthetic cache
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Follow-ups landed (3 commits on top of
|
| SHA | Scope |
|---|---|
| 582e079 | Copilot review fixes (A1-A7) -- exit-code contract + types + diagnostics + B3 normalization extract + B4 logging |
| 9e3354a | README link to drift-detection guide (B5) |
| 978765d | Cache-pin marker for stale-cache detection (B1) |
Cache-pin marker contract (B1, supply-chain follow-up)
apm install drops a .apm-pin JSON marker ({schema_version: 1, resolved_commit: <sha>}) into each cached package root. apm audit verifies it before drift replay. Catches the two real-world hazards the panel called out:
- Teammate bumps
apm.lock.yaml(e.g. pin X -> Y) without re-runningapm install-> drift would otherwise diff new lockfile against stale cache and report misleading findings. - Shared CI runner reuses
apm_modules/across builds where the lockfile changed between jobs.
LockfileBuilder syncs markers unconditionally (even when the lockfile YAML is unchanged AND when no install happens), so existing users self-heal on their next apm install. Threat model is documented in cache_pin.py -- this is stale-cache detection, NOT cryptographic integrity. Active-tampering defense (content-addressed hashes / signatures) is deferred.
Proof of integration test coverage
76 drift tests (was 74 prior to follow-ups; +14 unit cache_pin + new integration test):
| File | Count | Scenarios |
|---|---|---|
tests/unit/install/test_drift.py |
19 | Normalization (Build-ID, BOM, CRLF) + 3 inverse-norm regressions + CheckLogger stderr (incl. 2 new scratch_root verbose-gating tests) + replay correctness |
tests/unit/install/test_cache_pin.py |
14 | NEW -- positive verify, missing marker, malformed JSON, non-object payload, unsupported schema_version, missing resolved_commit field, commit mismatch, idempotent re-write, missing-dir tolerance, sync skips local deps, sync skips no-commit deps, sync skips no-cache deps, self-heal regression |
tests/integration/test_drift_check.py |
32 | 9 drift cases + 4 panel-feedback regressions + 12 edges + 5 multi-target + 5 default-behavior/--no-drift |
tests/integration/test_drift_check_e2e.py |
11 | E2E install-audit-tamper-reinstall loop + JSON/SARIF stable shape + ASCII-only output + bare-vs-CI exit-code regression + NEW cache-pin marker E2E (synthetic remote-style lockfile) |
uv run pytest tests/unit/install/test_drift.py \
tests/unit/install/test_cache_pin.py \
tests/integration/test_drift_check.py \
tests/integration/test_drift_check_e2e.py
============== 76 passed in 3.71s ==============
Plus the broader local slice ran clean: tests/unit/install/ tests/unit/policy/ tests/unit/utils/ + drift integration -> 1142 passed, 17 subtests passed.
Coverage of past + present drift hazards
| Hazard | Test reference |
|---|---|
| Build-ID line spuriously reported as drift | test_drift.py::test_normalize_strips_build_id_lines |
| Inverse: real change near a Build-ID line not masked | test_drift.py::test_normalize_does_not_mask_real_drift_near_build_id |
| BOM / CRLF spurious drift | test_drift.py::test_normalize_*_bom, *_crlf |
| Inverse: real change masked by BOM/CRLF strip | test_drift.py::test_normalize_does_not_mask_real_drift_* |
Bare apm audit exits non-zero on drift (regression) |
test_drift_check_e2e.py::test_bare_audit_with_drift_exits_zero_but_ci_audit_exits_one |
apm audit --ci does NOT gate on drift (regression) |
same |
| Stale cache after lockfile bump produces false drift | test_cache_pin.py (14 tests) + test_drift_check_e2e.py::test_apm_install_writes_cache_pin_marker_for_each_remote_dep |
| Cache pre-dating marker contract is never marked | test_cache_pin.py::test_sync_markers_self_heals_caches_missing_marker + integration counterpart |
| Marker tampering / corruption | test_cache_pin.py::test_verify_* (5 error paths) |
Lint
uv run --extra dev ruff check src/ tests/ -> All checks passed!
uv run --extra dev ruff format --check src/ tests/ -> 681 files already formatted
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 1 | 2 | Solid replay-engine pattern; _ReadOnlyProjectGuard is defined but never wired into run_replay() -- dead code. |
| CLI Logging Expert | 0 | 2 | 1 | scratch_root() stderr routing is correct (B4 resolved); modified-drift text color violates traffic light; 3 CacheMissError messages missing actionable fix. |
| DevX UX Expert | 0 | 2 | 1 | CacheMissError silently swallowed in bare apm audit; module docstring contradicts shipped feature. |
| Supply Chain Security Expert | 0 | 2 | 1 | _assert_scratch_bound bypasses canonical path guard API; resolved_commit=None silently skips cache-pin verification. |
| OSS Growth Hacker | 0 | 3 | 1 | Strong CI friction-reduction story; CHANGELOG reads as implementation prose, guide lacks a 60-second demo. |
| Doc Writer | 0 | 2 | 3 | Guide is PROSE-compliant; stale '7 baseline checks' in ci-cd.md/governance-guide.md understates new behavior; security.md has no drift mention. |
| Test Coverage Expert | 0 | 1 | 2 | Core drift scenarios and cache-pin error paths are well-covered; self-heal branch E2E reachability is unverifiable without running the test. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 5 follow-ups
- [Supply Chain Security Expert] Remote deps with
resolved_commit=Nonesilently skip cache-pin verification; add aCacheMissErroror visible warning for this case, plus a unit test proving the behavior. --outcome:missingon asecure-by-defaultsurface inherits near-blocking weight per CEO evidence rules; this is the highest-risk silent fail-open in the PR. - [Python Architect] Wire
_ReadOnlyProjectGuardintorun_replay()-- it is currently defined inguards.pybut never imported or called. -- The defense-in-depth no-write contract is dead code; the module docstring explicitly describes it as the intended wrapping pattern forrun_replay(). - [DevX UX Expert] Emit a stderr message when
CacheMissErroris swallowed in bareapm auditso users know drift was attempted and why it failed. -- Zero feedback on a stale-cache failure violates the failure-mode principle; confirmed by both devx-ux and cli-logging-expert independently. - [Doc Writer] Update
ci-cd.md,governance-guide.md, andsecurity.mdto replace '7 baseline checks' with the new default-on drift count, and fix the CHANGELOG relative-path link. -- Public doc surfaces understate shipped behavior; governance and security docs are the first place enterprise evaluators look. - [OSS Growth Hacker] Rewrite the CHANGELOG entry to lead with the user promise, add a 'Try it now' block to the drift-detection guide, and add a section heading to the before/after CI comparison. -- The strongest launch asset in this PR is currently buried mid-page; a 10-minute edit multiplies reach before the release note is cut.
Architecture
classDiagram
direction TB
class audit_command {
<<EntryPoint>>
+--ci bool
+--no-drift bool
}
class ReplayConfig {
<<ValueObject>>
+project_root Path
+lockfile_path Path
+targets frozenset
+cache_only bool
}
class DriftFinding {
<<ValueObject>>
+path str
+kind str
+package str
+inline_diff str
}
class CheckLogger {
+replay_start()
+diff_start()
+replay_complete(n int)
+clean()
+findings(n int)
}
class _ReadOnlyProjectGuard {
<<ContextManager>>
+project_root Path
+protected_roots list
+__enter__()
+__exit__()
}
class CacheMissError {
<<Exception>>
}
class CachePinError {
<<Exception>>
}
class run_replay {
<<IOBoundary>>
+config ReplayConfig
+logger CheckLogger
}
class diff_scratch_against_project {
<<Pure>>
+scratch_root Path
+project_root Path
}
class verify_marker {
<<IOBoundary>>
}
class write_marker {
<<IOBoundary>>
}
note for CheckLogger "Subclass pattern: redirects emit() to stderr so JSON/SARIF stdout stays clean"
note for _ReadOnlyProjectGuard "Defined in guards.py; NOT wired into run_replay() -- dead code gap (follow-up #2)"
note for run_replay "Replay Engine: cache-only materialization into scratch_root via integrate_package_primitives"
run_replay ..> ReplayConfig : reads
run_replay ..> CheckLogger : logs via
run_replay ..> verify_marker : stale-cache guard
run_replay ..> CacheMissError : raises
run_replay ..> diff_scratch_against_project : calls
diff_scratch_against_project ..> DriftFinding : returns list
verify_marker ..> CachePinError : raises on mismatch
_ReadOnlyProjectGuard ..> CacheMissError : raises
audit_command ..> run_replay : invokes via _check_drift
class ReplayConfig:::touched
class DriftFinding:::touched
class CacheMissError:::touched
class CachePinError:::touched
class CheckLogger:::touched
class _ReadOnlyProjectGuard:::touched
class run_replay:::touched
class diff_scratch_against_project:::touched
class verify_marker:::touched
class write_marker:::touched
classDef touched fill:#fff3b0,stroke:#d47600
flowchart TD
A([apm audit]) --> B{--ci?}
B -->|yes| C["_audit_ci_gate()\naudit.py:395"]
B -->|no| D["_audit_content_scan()\naudit.py:552"]
C --> C1["run_baseline_checks()\npolicy/ci_checks.py"]
C --> C2{not --no-drift\nand apm.yml?}
C2 -->|yes| DR["_check_drift()\npolicy/ci_checks.py"]
C2 -->|no| C3["[stderr] drift skipped"]
D --> D1["[FS] scan_lockfile_packages()"]
D --> D2{not --no-drift and\napm.yml and no --file/--strip?}
D2 -->|yes| DR
D2 -->|no| D3["[stderr] drift skipped"]
DR --> RP["run_replay(ReplayConfig)\ndrift.py:374"]
RP --> RP1["[FS] _make_scratch_root()\ntempfile.mkdtemp + atexit.register"]
RP --> RP2["_assert_scratch_bound()\npoint-in-time only"]
RP --> RP3{for each lock_dep\nin LockFile}
RP3 --> RP4["[FS] _materialize_install_path()\ndrift.py:198"]
RP4 --> RP5["[FS] verify_marker(candidate, commit)\ncache_pin.py -- CachePinError->CacheMissError"]
RP5 --> RP6["[FS] integrate_package_primitives()\nservices.py scratch_root=scratch"]
RP6 --> RP3
RP3 -->|all done| DIFF["diff_scratch_against_project()\ndrift.py:544"]
DIFF --> FIND["list[DriftFinding]\nmodified / unintegrated / orphaned"]
FIND --> EC{--ci mode?}
EC -->|yes| EXIT1(["sys.exit 0 or 1\ndrift escalates in --ci"])
EC -->|no| EXIT0(["sys.exit 0\ndrift advisory only"])
sequenceDiagram
actor User
participant CLI as audit.py
participant DriftChk as ci_checks._check_drift
participant Replay as drift.run_replay
participant Pin as cache_pin.verify_marker
participant Svc as services.integrate_package_primitives
participant Diff as drift.diff_scratch_against_project
User->>CLI: apm audit [--ci] [--no-drift]
CLI->>DriftChk: _check_drift(project_root, lockfile, cache_only=True)
DriftChk->>Replay: run_replay(ReplayConfig)
Replay->>Replay: _make_scratch_root() -- tempfile.mkdtemp
Replay->>Replay: _assert_scratch_bound() -- point-in-time check
loop for each lock_dep
Replay->>Pin: verify_marker(install_path, resolved_commit)
alt marker ok
Pin-->>Replay: ok
else CachePinError
Pin-->>Replay: raise CachePinError
Replay-->>DriftChk: raise CacheMissError
end
Replay->>Svc: integrate_package_primitives(scratch_root=scratch)
Svc-->>Replay: files written to scratch only
end
Replay-->>DriftChk: scratch_root Path
DriftChk->>Diff: diff_scratch_against_project(scratch, project, lockfile)
Diff-->>DriftChk: list[DriftFinding]
DriftChk-->>CLI: (CheckResult, drift_findings)
alt --ci mode
CLI->>User: render + sys.exit(1) if drift
else bare apm audit
CLI->>User: render (advisory) + sys.exit(0)
end
Recommendation
Merge now. The PR is well-tested, the primary safety guard is in place, and no panelist found a blocking defect. File three follow-up issues immediately post-merge: (1) wire _ReadOnlyProjectGuard + emit stderr on CacheMissError swallow (paired, same PR); (2) add resolved_commit=None warning/error with a secure-by-default unit test; (3) doc pass for stale check-count copy and CHANGELOG link. Track the growth-hacker content edits alongside the doc pass -- they are cheap and the launch window is now.
Full per-persona findings
Python Architect
-
[recommended]
_ReadOnlyProjectGuardinguards.pyis defined but never imported or called byrun_replay()atsrc/apm_cli/utils/guards.py:60
The module docstring onguards.pyexplicitly states it is the defense-in-depth check for drift replay and shows a usage example wrappingrun_replay(). Greppingsrc/confirms_ReadOnlyProjectGuardhas zero import or call sites outside its own file. The primary protection is_assert_scratch_bound()(drift.py:108), which is a point-in-time check at scratch dir creation -- it does NOT detect accidental real-project writes that could occur later. Leaving the guard unconnected means the no-write contract is enforced by convention only, not by a runtime assertion.
Suggested: Inrun_replay()(drift.py:424), wrap the inner replay loop:with _ReadOnlyProjectGuard(project_root, ['.apm', 'apm.lock.yaml', '.github']): for lock_dep in lock.get_all_dependencies(): ... -
[nit]
_inline_diff_for()returns empty string for in-range files;DriftFinding.inline_diffis always empty in practice atsrc/apm_cli/install/drift.py:532
drift.py:532-541defines_inline_diff_forbut only handles the too-large case.DriftFinding.inline_diffis therefore always''or the size note, never actual diff content. -
[nit]
normalization.pyexports underscore-prefixed private names in__all__atsrc/apm_cli/utils/normalization.py:50
__all__includes_BOM,_normalize,_normalize_line_endings,_strip_bom,_strip_build_id-- private-by-convention names. Mixed signals to future contributors and static analysis tools.
CLI Logging Expert
-
[recommended] Modified drift entries rendered yellow in text mode; traffic light rule requires red for error-severity findings at
src/apm_cli/utils/diagnostics.py:462
diagnostics.py _render_drift_grouprenders modified, unintegrated, and orphaned drift items all withcolor='yellow'. The SARIF renderer correctly classifies modified aslevel='error', but the text/diagnostic path does not distinguish it. Hand-edited deployed files (modified) are the highest-severity drift case and should use red.
Suggested:color = 'red' if label == 'modified' else 'yellow' -
[recommended] Three
CacheMissErrorraises lack 'run apm install' actionable text atsrc/apm_cli/install/drift.py:221
Lines 221, 225 (drift.py) and 397 surface as user-visible errors but give no next step. Lines 232-234 and 246 correctly include 'run apm install'. Inconsistency means users hit the first two local-source errors and have no guidance.
Suggested: Lines 221/225: append'-- run apm install'. Line 397: append'; run apm install to regenerate it'. -
[nit]
--no-driftmutexUsageErrormessage explains why but omits the fix atsrc/apm_cli/commands/audit.py:884
Current message names the flags but does not tell the user what to do instead.
Suggested: Add:'Run each mode separately: apm audit --no-drift for content scan, apm audit for drift detection.'
DevX UX Expert
-
[recommended]
CacheMissErrorfrom drift replay is silently swallowed in bareapm audit; user gets no output atsrc/apm_cli/commands/audit.py:676
In_audit_content_scan, when_check_driftraisesCacheMissErrorit returns(CheckResult(passed=False, message='...'), []).drift_findings=[]so theif drift_findings:branch never fires, anddrift_failedis explicitly discarded. The user gets zero feedback that drift was attempted and failed due to a stale cache. Violates: "Failure mode is the product."
Suggested: Afterdrift_check, drift_findings = _check_drift(...), checkif not drift_check.passed and not drift_findings:and emit the check message to stderr. -
[recommended] Module docstring (lines 5-6) says drift detection is 'planned as future modes' and names the flag
--drift; both are wrong after this PR atsrc/apm_cli/commands/audit.py:5
The docstring reads: "lock-file consistency (--ci) and drift detection (--drift) are planned as future modes." This PR ships drift as default-on with--no-drift(not--drift), making the docstring doubly wrong.
Suggested: Replace with accurate description of the shipped behavior. -
[nit] Mutex error message reads as self-defeating: 'those modes do not run drift detection' implies
--no-driftis redundant atsrc/apm_cli/commands/audit.py:884
Confusing for users who wonder why the error fires at all.
Supply Chain Security Expert
-
[recommended]
_assert_scratch_boundusesPath.relative_to()directly instead of the canonicalensure_path_withinguard atsrc/apm_cli/install/drift.py:108
The persona contract requires every path-containment check to flow throughensure_path_withinso symlink resolution, Windows extended-prefix stripping, and single-choke-point invariant are consistently applied._assert_scratch_boundcallsproject_root.resolve() / scratch_root.resolve()thenPath.relative_to()-- equivalent in practice but outside the canonical chokepoint. Ifensure_path_within's logic ever tightens,_assert_scratch_boundsilently diverges.
Suggested: Addassert_path_outside(child: Path, base: Path)topath_security.py, then replace_assert_scratch_bound's body with a call to it. -
[recommended]
resolved_commit=None/empty silently skips cache-pin verification in_materialize_install_pathatsrc/apm_cli/install/drift.py:240
Lines 240-246:verify_markeris gated onif lock_dep.resolved_commit:. A lockfile entry whoseresolved_commitisNoneor empty string bypasses stale-cache detection entirely and proceeds to diff against whatever is inapm_modules-- silently. For remote deps, this should emit a visible warning or raiseCacheMissError.
Suggested: For remote deps (source != 'local'), treat a missingresolved_commitas aCacheMissErroror emit a visible warning.
Proof (missing at unit):tests/unit/install/test_drift.py::test_remote_dep_none_commit_warns_or_raises-- proves: A remote dep with noresolved_commitcauses aCacheMissErroror explicit warning rather than silently skipping cache-pin verification [secure-by-default] -
[nit]
sync_markers_for_lockfileuses bareexcept Exceptionswallowing all install-path resolution errors atsrc/apm_cli/install/cache_pin.py:152
Line 152 catches all exceptions to keep marker sync best-effort, but also masksAttributeError/TypeErrorfrom programming mistakes.
Suggested: Narrow to(AttributeError, ValueError, OSError)and log atDEBUGlevel.
OSS Growth Hacker
-
[recommended] CHANGELOG entry is implementation prose, not a user story; release-note audience needs the benefit first at
CHANGELOG.md:12
The first bullet leads with flag semantics (JSON keys, SARIF rule IDs, mutual-exclusion rules) before naming the user benefit. Existing users scanning the changelog to decide whether to upgrade need 'you no longer need bash glue in CI' in the first sentence.
Suggested: Open with: 'apm audit now catches missed installs, hand-edits, and orphaned files in one command -- no bash glue needed. CI gates drop from a 5-line git-status script toapm audit --ci.' Then append the flag/schema detail. -
[recommended] Drift detection guide has no 60-second runnable demo; a reader cannot feel the feature without an existing project at
docs/src/content/docs/guides/drift-detection.md
The guide explains the feature well but never gives a reader a copy-paste block they can run right now and see real output.
Suggested: Add a '## Try it now' section:git clone https://github.com/microsoft/apm && cd apm && apm install && apm audit -
[recommended] The CI before/after is a quotable launch beat but buried mid-page with no skimmable heading at
docs/src/content/docs/guides/drift-detection.md:102
The before/after code block (5-line bash -> singleapm audit --ci) is the strongest friction-reduction proof. Currently it appears at line 102 with no named callout, no framing sentence, and no section heading.
Suggested: Rename 'CI integration' to 'Before and after: CI gate in one line' and add a lead sentence. -
[nit] README bullet 'catch hand-edits before they ship' narrows the hook to one of three drift kinds at
README.md:84
Users who forget to re-run install (unintegrated) may not self-identify as 'hand-editors'.
Suggested: 'catches missed installs, hand-edits, and orphaned files before they ship'
Auth Expert -- inactive
No auth-relevant surface touched; drift detection is cache-only with no network calls, auth resolution, token management, or credential handling.
Doc Writer
-
[recommended]
ci-cd.mdandgovernance-guide.mdstill describeapm audit --cias running '7 baseline checks' -- drift is now a default-on eighth check atdocs/src/content/docs/integrations/ci-cd.md:150
ci-cd.mdandgovernance-guide.mdrepeat the '7 baseline' count. Since drift detection runs by default in--ciand contributes to exit code, every one of these statements now understates the behavior.
Suggested: Change to '7 baseline lockfile checks plus integration drift detection (default-on), no configuration'. Updategovernance-guide.mdsimilarly. -
[recommended]
enterprise/security.mddescribesapm auditonly as a content-scanning tool; drift detection is not mentioned atdocs/src/content/docs/enterprise/security.md:130
Per the two-layer security model, all security-related sections must describeapm auditcorrectly.security.mdenumeratesapm auditcapabilities with no mention of drift.
Suggested: Add a cross-reference to the Drift Detection guide after the existingapm auditbullet list. -
[nit] CHANGELOG link uses a repo-relative file path, not a rendered docs URL -- inconsistent with all other entries at
CHANGELOG.md:12
Every other hyperlink in the CHANGELOG useshttps://external URLs. A file path is not clickable from a rendered GitHub CHANGELOG view.
Suggested: Replace with `(microsoft.github.io/redacted) -
[nit]
drift-detection.mdexit code table:--no-driftrow value 'governed only by other checks' is ambiguous atdocs/src/content/docs/guides/drift-detection.md:55
The other two rows give explicit values (0, 1). The vague phrase breaks the table's parallelism.
Suggested: '0 (advisory) or 1 with--ci-- same as baseline checks without drift' -
[nit]
drift-detection.mdflowchart names 'scratch tmpdir' -- raises a question about cleanup guarantees not addressed in the guide atdocs/src/content/docs/guides/drift-detection.md:33
'tmpdir' implies a system temp directory, raising a question about whether cleanup is guaranteed.
Suggested: 'Replay install into scratch tree (auto-cleaned)'
Test Coverage Expert
-
[recommended]
LockfileBuilder._sync_cache_pin_markers_from_diskself-heal branch lacks a confirmed explicit integration test
The E2E testtest_apm_install_writes_cache_pin_marker_for_each_remote_depmay or may not exercise the_sync_cache_pin_markers_from_diskbranch (vs the in-memory path). Without running the test it is not possible to confirm which branch fires. If the self-heal path is unexercised, a future refactor could silently break the upgrade story.
Suggested: Add a dedicated integration test: install a project, wipe all.apm-pinmarkers, runapm installwithout changingapm.yml, assert markers are re-created.
Proof (unknown at e2e):tests/integration/test_drift_check_e2e.py::TestDriftE2E::test_apm_install_writes_cache_pin_marker_for_each_remote_dep-- proves:apm installself-heals.apm-pinmarkers for pre-existing caches [secure-by-default, devx]
assert marker.exists(), 'install must self-heal pre-existing caches by writing the marker' -
[nit]
diagnostics._render_drift_groupemits no remediation hint; no test guards against that omission attests/integration/test_drift_check.py:711
No test asserts a recovery hint ('run apm install') in drift text output.
Suggested: Addassert 'apm install' in combinedtotest_e9_text_mode_drift_summary_includes_kind_and_path.
Proof (missing at integration-with-fixtures):tests/integration/test_drift_check.py::TestSectionENoDriftFlag::test_e9_text_mode_drift_summary_includes_kind_and_path
assert 'apm install' in combined # recovery action must be present in drift output -
[nit]
services.pyscratch_root guard rejection path has no unit test attests/unit/install/test_drift.py
Greppedtests/unit/andtests/integration/forscratch_root; no match covering the rejection case (project_root escaping scratch_root).
Suggested: Add a unit test assertingPathSecurityErroris raised whenproject_rootpoints outsidescratch_root.
Proof (missing at unit):tests/unit/install/test_drift.py::test_scratch_root_guard_rejects_project_root_escaping_scratch
with pytest.raises(PathSecurityError): run_drift_replay(project_root=outside, scratch_root=scratch)
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Note
🔒 Integrity filter blocked 1 item
The following item was blocked because it doesn't meet the GitHub integrity level.
- feat(audit): default-on integration drift detection #1137
pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
To allow these resources, lower min-integrity in your GitHub frontmatter:
tools:
github:
min-integrity: approved # merged | approved | unapproved | noneGenerated by PR Review Panel for issue #1137 · ● 5.8M · ◷
|
Acknowledging the follow-up batch from the panel synthesis (#issuecomment-4377781459). Working through these in disciplined order:
Each will be implemented via the corresponding specialist agent (with test-coverage cross-validation on C1–C3). Posting progress as commits land. |
|
(read-only) |
C1 (supply-chain): Fail closed on unpinned remote deps - cache_pin.find_unpinned_remote_deps() helper + stderr warning in sync_markers_for_lockfile - drift._materialize_install_path raises CacheMissError for remote deps with resolved_commit=None (was silent fail-open) - Replaced silent-skip test with warning assertion + new helper test C2 (architecture): Wire _ReadOnlyProjectGuard into run_replay - run_replay() now wraps the deps loop with _ReadOnlyProjectGuard on governed root dirs + apm.lock.yaml + AGENTS.md - Regression test: monkeypatched leaky integrator triggers ProtectedPathMutationError C3 (cli-logging-ux): Stderr message on swallowed CacheMissError - audit._audit_content_scan emits '[!] drift check could not run: <msg>' to stderr when drift_failed and no findings (covers cache miss, missing lockfile, cache-pin error) - Integration test e10 asserts stderr message in bare-audit path C4 (docs): Baseline-check phrasing + CHANGELOG link - governance-guide, ci-cd, cli-commands now read '7 baseline checks plus integration drift detection' - CHANGELOG drift-detection link points to docs site URL C5 (oss-growth): User-promise framing - CHANGELOG drift entry leads with the user promise (forgotten installs + hand-edits) before mechanism - drift-detection.md gains a 'Try it now' block at the top - Before/after CI comparison promoted to its own subsection with explicit framing of what the bash workaround missed Verification: ruff check + format silent; 7621 unit tests + 27 drift integration tests green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Panel follow-ups C1-C5 addressed (commit 60bf38a)All five items from the panel review are implemented per the specialist guidance, with cross-validation across the relevant areas. C1 -- Supply-chain: fail closed on unpinned remote depsSpecialist: supply-chain-security Changes:
Tests: C2 -- Architecture: wire
|
Collapse the two added entries (drift + cache-pin markers) into one short line that answers the developer 'so what?' and points to the Drift Detection guide for the full mechanism + opt-out + cache-pin details. Per maintainer feedback: the previous entries were too long for a CHANGELOG. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Roll forward the four PRs merged since v0.12.1: - #1137 feat(audit): default-on integration drift detection - #1135 fix(deps): subdir-agnostic bare cache (parallel sparse-checkout race) also resolves duplicate report #1140 (ADO sub-path manifestation) - #1129 docs: first-package guide accuracy (includes: auto, skill paths) - #1127 docs: APM's role for skills, plugins-as-packages, ADO sub-paths Bump pyproject.toml + uv.lock 0.12.1 -> 0.12.2 and convert the Unreleased CHANGELOG block into the 0.12.2 section, with a single 'so what' line per merged PR per the changelog contract. Co-authored-by: Daniel Meppiel <copilot-rework@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ocks v0.12.2 release) (#1142) * fix(install): exclude .apm-pin marker from package content hash Re-installing any package previously installed by APM 0.12.2 erased its apm_modules/ directory: the .apm-pin cache marker (added by #1137 for drift-replay) is written AFTER hash recording, so the on-disk hash diverged from the lockfile hash on every subsequent install, falsely tripping the supply-chain content-hash mismatch check and rmtree'ing the cache. Surfaced by tests/integration/test_guardrailing_hero_e2e.py, which runs 'apm install <virtual-file>' twice via run_command(show_output= True) and asserts apm_modules/github/<flattened-name> still exists -- the v0.12.2 release tag failed in CI on every platform on this test. Fix: exclude .apm-pin from compute_package_hash via a new _EXCLUDED_FILES set, mirroring _EXCLUDED_DIRS. Add regression test test_skips_apm_pin_marker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(install): scope .apm-pin exclusion to package root Address Copilot review feedback on PR #1142: a basename match would exclude any .apm-pin file at any depth, so a malicious package could hide bytes under subdir/.apm-pin to bypass the integrity hash. The cache-pin marker is only ever written to the package root by cache_pin.write_marker, so scope the exclusion accordingly. Extend the regression test to assert nested .apm-pin files are still hashed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Daniel Meppiel <copilot-rework@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
feat(audit): default-on integration drift detection
TL;DR
apm auditnow detects integration drift by default. It replaysapm installcache-only into a scratch tree and diffs against your working copy, catching the three cases that previously slipped past CI:.apm/source added without re-running install (#1067), hand-edited deployed files, and orphan files left after a dependency was removed. The scan is read-only — nothing in the project, lockfile, orapm_modules/is touched. A new.apm-pincache marker, written byapm install, defends drift from the stale-cache failure mode (lockfile bumped → cache not refreshed).Note
Closes #1071. Closes #898 — the integration-drift rail completes the epic; the per-file hash rail was delivered by #762/#889, and cross-skill dependency closure is already supported via the existing transitive resolver (skill bundles ship their own
apm.yml). Review-cycle status: all Copilot inline comments (A1–A7) and apm-review-panel deferred follow-ups (B1 cache-pin, B2 inverse-norm, B3 normalization extract, B4 stderr logging, B5 README link) are landed in this PR.Problem (WHY)
Today,
apm auditcovers lockfile consistency but is blind to the most common real-world drift mode: a developer adds a.apm/instructions/foo.mdand forgets to re-runapm install, so.github/instructions/foo.mdnever lands. CI only catches it because every consumer repo (including this one) maintains a hand-rolledgit status --porcelainscript after the install step. That workaround has three problems:apm installfirst, so localapm auditgives false confidence.unintegrated(forgot to install) frommodified(hand-edit) fromorphaned(stale file), so the failure message is unhelpful.apm audit, not as a side-channel script.Issue #898 originally asked for full lockfile-vs-deployed-content verification; the design loop captured in
WIP/drift/widened the scope to all three drift kinds and locked the surface as a single command (apm audit) with default-on behavior.Approach (WHAT)
apm audit(no new command)--no-driftmutually exclusive with--stripand--filedriftkey in JSON;apm/drift/rules in SARIF--cionlyapm auditis advisory; CI mode is the gate.apm-pinJSON marker per cached package, verified before replayImplementation (HOW)
src/apm_cli/install/drift.py(new, ~430 lines)ReplayConfig,DriftFinding,CacheMissError,CheckLogger(withscratch_root()stderr helper),run_replay(),diff_scratch_against_project(), three renderers (text/JSON/SARIF). Reads.apm-pinmarker viaverify_marker()before each cache-only materializationsrc/apm_cli/install/cache_pin.py(new)MARKER_FILENAME=".apm-pin",SCHEMA_VERSION=1,write_marker,verify_marker,sync_markers_for_lockfile,CachePinErrorsrc/apm_cli/install/phases/lockfile.pyLockfileBuilder.build_and_save()calls_sync_cache_pin_markersunconditionally after_write_if_changedAND_sync_cache_pin_markers_from_disk()on the no-install early-return path (self-heals existing caches on nextapm install)src/apm_cli/utils/normalization.py(new)drift.pyfor back-compatsrc/apm_cli/install/services.pyscratch_rootkwarg tointegrate_package_primitives()so replay can redirect deploy paths without monkey-patchingsrc/apm_cli/utils/guards.py(new)_assert_scratch_bounddefensive guard preventing replay from ever writing outside the scratch treesrc/apm_cli/utils/diagnostics.pyDRIFTdiagnostic category with friendly remediation text (DRIFT_ORPHANEDclarifies it is a remediation hint)src/apm_cli/commands/audit.py--no-driftflag, mutex viaUsageError, drift wired into both_audit_ci_gateand_audit_content_scan. Bareapm auditis advisory (exit 0);apm audit --ciis the gate (exit 1)src/apm_cli/policy/ci_checks.py_check_driftsignature; CacheMiss wording is drift-specific.github/workflows/ci.ymlapm-actionships with this CLI; kept as defense-in-depth fallback for one release cycletests/integration/test_drift_check.py(new, 32 tests)--no-drift/exit-codestests/integration/test_drift_check_e2e.py(new, 12 tests)tests/unit/install/test_drift.py(new, 19 tests)tests/unit/install/test_cache_pin.py(new, 14 tests)tests/unit/install/test_drift_perf.py(new, 1 test)CHANGELOG.md,docs/.../guides/drift-detection.md,packages/apm-guide/.apm/skills/apm-usage/commands.md, README link to drift guideDiagrams
The drift pipeline at runtime — cache-only replay with marker verification, then diff:
flowchart LR A["apm audit"] --> B["Read apm.lock.yaml<br/>+ persistent cache"] B --> P["verify_marker():<br/>.apm-pin matches<br/>resolved_commit?"] P -->|"yes"| C["run_replay():<br/>install pipeline into<br/>scratch tmpdir"] P -->|"no"| X["CacheMissError:<br/>run apm install"] C --> D["diff_scratch_against_project():<br/>scratch vs project"] D --> E["DriftFinding[]"] E --> F["render_drift_text /<br/>render_drift_json /<br/>render_drift_sarif"]How drift composes with the existing audit checks (drift is a separate rail, not a fake check):
stateDiagram-v2 [*] --> AuditEntry AuditEntry --> CIGate: --ci AuditEntry --> ContentScan: bare or PKG CIGate --> Baseline7: 7 lockfile checks Baseline7 --> Drift: if --no-drift not set ContentScan --> Drift: if --no-drift not set Drift --> Render Baseline7 --> Render Render --> Exit1: any failure in --ci Render --> Exit0: bare audit (advisory)Trade-offs
--no-drift. Rejected: opt-in default — would replicate the fix(compile): emit and clean up copilot root instructions (#792) #1067 silent-failure mode this PR exists to fix.--strip/--fileis strict. A user could theoretically want strip-mode + drift, but the semantics ("drift of what?") are undefined. Refused at the CLI rather than silently picked. Rejected: warn-and-continue — invites bug reports we cannot answer..apm-pinmarker proves "the lockfile that produced this cache matches the lockfile we're auditing"; it does not defend against an active adversary with write access toapm_modules/. Content-addressed hashes / signatures are deferred to a follow-up. Documented incache_pin.pymodule docstring.apm auditmode. In--cimode the lockfile-exists check fires first and fails loudly; bare mode no-ops drift. Pinned by tests; tracked as follow-up.apm.yml'starget:field. Couples engine to manifest format, but directory auto-detection alone missed targets whose deployment dirs were still empty (false orphan reports). Caught during Phase D test development.Benefits
unintegrated,modified,orphaned) that previously needed three separate scripts.apm audit --cisubsumes both lockfile fidelity and integration drift; the bashgit status --porcelainscript is now annotated as redundant..apm-pinmarker +verify_markerrejects a cache that doesn't match the lockfile;apm installself-heals existing caches without user intervention.Validation
Scenario evidence — what is tested to work
Every user-promise scenario this PR introduces is mapped to the test that proves it. Principle taxonomy per the scenario-evidence rubric.
.apm/source — drift caughttest_drift_check.py::TestSectionADriftCases::test_a1_unintegrated_local_source_detected.github/file — drift caughttest_drift_check.py::TestSectionADriftCases::test_a2_modified_deployed_file_detectedtest_drift_check.py::TestSectionADriftCases::test_a3_orphaned_deployed_file_detectedtest_drift_check.py::TestSectionBNormalization::test_b1_build_id_only_diff_no_drifttest_drift_check.py::TestSectionBNormalization::test_b3_crlf_only_diff_no_drifttest_drift_check.py::TestSectionBNormalization::test_b2_bom_only_diff_no_drifttest_drift_check.py::TestSectionDMultiTarget::test_d1_multi_target_no_drift_when_clean--no-driftopt-out skips drift entirelytest_drift_check.py::TestSectionENoDriftFlag::test_e3_no_drift_flag_skips_replay--no-driftmutex with--strip/--filerejects via UsageErrortest_drift_check.py::TestSectionENoDriftFlag::test_e5_no_drift_mutex_with_strip_rejected/test_e6_no_drift_mutex_with_file_rejectedapm audit --ciexits 1 on drifttest_drift_check.py::TestSectionENoDriftFlag::test_e1_drift_in_ci_mode_exits_oneapm auditreports drift but exits 0 (advisory)test_drift_check_e2e.py::TestDriftE2E::test_bare_audit_with_drift_exits_zero_but_ci_audit_exits_onedriftkeytest_drift_check_e2e.py::TestDriftE2E::test_audit_ci_json_payload_has_stable_top_level_keysapm/drift/rulestest_drift_check_e2e.py::TestDriftE2E::test_audit_ci_sarif_payload_is_valid_sarifapm_modules/test_drift_check_e2e.py::TestDriftE2E::test_apm_audit_makes_no_writes_to_working_tree+test_apm_audit_makes_no_writes_when_drift_presenttest_drift_check_e2e.py::TestDriftE2E::test_audit_runs_without_network_subprocessestest_drift_check_e2e.py::TestDriftE2E::test_install_audit_tamper_audit_reinstall_audit_looptest_drift_check_e2e.py::TestDriftE2E::test_audit_completes_within_smoke_budgettest_drift_perf.py::test_drift_replay_100_primitives_completes_within_budgettest_drift_check_e2e.py::TestDriftE2E::test_audit_text_mode_drift_section_uses_ascii_onlyapm installwrites.apm-pinmarker for each remote deptest_drift_check_e2e.py::TestDriftE2E::test_apm_install_writes_cache_pin_marker_for_each_remote_dep.apm-pindoesn't match the lockfile committest_cache_pin.py::test_verify_marker_raises_on_commit_mismatch(unit) + composes with #20 (e2e)Scenario evidence — what is tested NOT to regress
Each row is a real bug or false-positive that escaped to production at some point in the past. The regression-trap test is the assertion that, had it existed before, would have caught the bug. No silent regression on any of these is permitted.
.apm/source added under a non-default subdir (Cursor commands),.cursor/commands/never integratedtest_drift_check.py::TestSectionCRegressions::test_c1_regression_pr_1067_unintegrated_subdir_sourceapm.ymltarget:field, falsely orphaning filestest_drift_check.py::TestSectionCRegressions::test_c2_regression_pr_882_target_autodetection_alignmenttest_drift_check.py::TestSectionCRegressions::test_c3_regression_pr_889_orphan_cleanup_does_not_touch_governedapm installnot re-run, deploys stayedtest_drift_check.py::TestSectionCRegressions::test_c4_regression_source_deleted_no_installtest_drift.py::test_normalize_strips_build_id_lines+ inversetest_normalize_does_not_mask_real_drift_near_build_idtest_drift.py::test_normalize_strips_*_bom,*_crlf+ inversetest_normalize_does_not_mask_real_drift_*apm audit --cidid not gate on drift findingstest_drift_check_e2e.py::TestDriftE2E::test_bare_audit_with_drift_exits_zero_but_ci_audit_exits_oneapm auditexited non-zero on drift, breaking advisory contractapm.lock.yamlbut didn't re-runapm install; drift compared new lockfile against stale cachetest_cache_pin.py(14 tests covering verify, sync, self-heal) +test_drift_check_e2e.py::test_apm_install_writes_cache_pin_marker_for_each_remote_deptest_cache_pin.py::test_sync_markers_self_heals_caches_missing_markertest_cache_pin.py::test_verify_marker_raises_on_*(5 tests)test_drift.py::test_check_logger_scratch_root_emits_to_stderr_when_verbose+test_check_logger_scratch_root_silent_when_not_verboseThe CI integration script (
scripts/test-integration.sh) was extended with two newpytestblocks inrun_e2e_tests()so both integration suites run in the e2e CI job.How to test
uv sync --all-extras --devuv run --extra dev pytest tests/unit/install/test_drift.py tests/unit/install/test_cache_pin.py tests/unit/install/test_drift_perf.py tests/integration/test_drift_check.py tests/integration/test_drift_check_e2e.py -v— expect 77 passedapm.yml), edit any file under.github/instructions/and runapm audit— expect amodifiedfinding, exit 0apm audit --no-drift— drift section absent, lockfile checks still runapm audit --ci -f json— JSON has top-leveldriftkey, parses cleanly, exit code is 1 if drift existsNote
The
.github/workflows/ci.ymllegacy bash drift check is intentionally kept for one release cycle as defense-in-depth untilapm-actionpicks up a CLI version that includes this PR.Closes #1071. Closes #898.
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com